Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | Go Back
View Post [edit]

Poster: SethPen Date: Oct 1, 2013 9:19am
Forum: web Subject: "retroactive and permanent"

Greetings,

I have a site that I like to view from time to time and I am concerned, with the current rules, it may not be available for future individuals.


http://portal.stodge.org (Page cannot be crawled or displayed due to robots.txt.)

From your Frequently Asked Questions:

" Currently there is no way to exclude only a portion of a site, or to exclude archiving a site for a particular time period only. When a URL has been excluded at direct owner request from being archived, that exclusion is retroactive and permanent. "


Previously, the owner wanted this to be Archived:

http://everything2.com/title/Portal%253A+A+Dataspace+Retrieval?author_id=888590#salimfadhley

(This is an online version of a book called "Portal: A Dataspace Retrieval")

"The pages are designed to be search engine friendly – should a crawler bump into one of the pages the entire site should be absorbed in no time at all. I hope people will soon start bumping into theses pages at random, and then perhaps start reading this odd story. "


Am I to understand correctly from your Frequently Asked Questions, that if a site posts a new robot.txt with the correct parameters to not be index, it will be retroactive? Including previous years?

Will this ever change?

Additionally I want to thank you for a great service and I have been amazed with what you have been able to archive.